feat: Airflow 3.0.1 (experimental) #630

adwk67 · 2025-05-20T10:02:30Z

Description

Related to #618

Most changes in this PR are due to the fact that the API has changed and both old and new versions need to be available for the tests. Code changes can be summarised as:

adding a condition for the start commands
adding a condition for new env-vars
minor change to logging definition

N.B. this PR does not cover updates to OPA-Auth (covered separately).

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
~~CRD changes approved~~
~~CRD documentation for all fields, following the [style guide]~~
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Roadmap has been updated

maltesander

First batch, will do some more testing.

maltesander · 2025-05-27T13:02:16Z

rust/operator-binary/src/crd/mod.rs

+                min: Some(Quantity("1".into())),
                max: Some(Quantity("2".into())),
            },
            MemoryLimitsFragment {
-                limit: Some(Quantity("2Gi".into())),
+                limit: Some(Quantity("3Gi".into())),
                runtime_limits: NoRuntimeLimitsFragment {},
            },
        ),
        AirflowRole::Webserver => (
            CpuLimitsFragment {
-                min: Some(Quantity("500m".into())),
+                min: Some(Quantity("1".into())),
                max: Some(Quantity("2".into())),


There is a docs part for default ressources (outdated) https://github.com/stackabletech/airflow-operator/blob/feat/airflow-3.0.0/docs/modules/airflow/pages/usage-guide/storage-resources.adoc that could be updated :)

maltesander · 2025-05-27T13:20:31Z

rust/operator-binary/src/env_vars.rs

+        env.insert("AIRFLOW__API_AUTH__JWT_SECRET".into(), EnvVar {
+            name: "AIRFLOW__API_AUTH__JWT_SECRET".into(),
+            value: Some(JWT_KEY.clone()),
+            ..Default::default()
+        });


This will change if the operator restarts? That will invalidate access tokens, user sessions etc.? Maybe it makes sense to auto generate (or provide) a secret? Similar to the nifi sensitive properties key ?

rust/operator-binary/src/crd/mod.rs

maltesander · 2025-05-28T13:52:50Z

rust/operator-binary/src/crd/mod.rs

+                    command.extend(vec![
+                        "prepare_signal_handlers".to_string(),
+                        format!("containerdebug --output={STACKABLE_LOG_DIR}/containerdebug-state.json --loop &"),
+                        "airflow api-server &".to_string(),
+                    ]);


This could be templated for the process (api-server, celery worker...) and reused?

maltesander · 2025-05-28T14:24:24Z

tests/templates/kuttl/external-access/install-airflow-cluster.yaml.j2

+{% if test_scenario['values']['airflow-latest'].find(",") > 0 %}
+    custom: "{{ test_scenario['values']['airflow-latest'].split(',')[1] }}"
+    productVersion: "{{ test_scenario['values']['airflow-latest'].split(',')[0] }}"


Is it on purpose to only test this (and others) for 3.x?

that one should probably be for all airflow versions, yes, as it is new functionality.

maltesander · 2025-05-28T14:41:46Z

tests/templates/kuttl/logging/51-assert.yaml.j2

+{% if test_scenario['values']['airflow'].find(",") > 0 %}
+  - script: |
+      kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group automatic-log-config --airflow-version "{{ test_scenario['values']['airflow'].split(',')[0] }}"
+      kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group custom-log-config --airflow-version "{{ test_scenario['values']['airflow'].split(',')[0] }}"
+{% else %}
+  - script: |
+      kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group automatic-log-config --airflow-version "{{ test_scenario['values']['airflow'] }}"
+      kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group custom-log-config --airflow-version "{{ test_scenario['values']['airflow'] }}"
+{% endif %}


Not much nicer, but less repetition (could be used elsewhere as well) :)

Suggested change

{% if test_scenario['values']['airflow'].find(",") > 0 %}

- script: |

kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group automatic-log-config --airflow-version "{{ test_scenario['values']['airflow'].split(',')[0] }}"

kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group custom-log-config --airflow-version "{{ test_scenario['values']['airflow'].split(',')[0] }}"

{% else %}

- script: |

kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group automatic-log-config --airflow-version "{{ test_scenario['values']['airflow'] }}"

kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group custom-log-config --airflow-version "{{ test_scenario['values']['airflow'] }}"

{% endif %}

{% if test_scenario['values']['airflow'].find(",") > 0 %}

{% set airflow_version = test_scenario['values']['airflow'].split(',')[0] %}

{% else %}

{% set airflow_version = test_scenario['values']['airflow'] %}

{% endif %}

- script: |

kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group automatic-log-config --airflow-version "{{ airflow_version }}"

kubectl exec -n $NAMESPACE test-airflow-python-0 -- python /tmp/health.py --role-group custom-log-config --airflow-version "{{ airflow_version }}"

adwk67 and others added 6 commits May 16, 2025 16:10

wip: working tests (except auth: opa, oidc)

f57633a

wip: changing python test files

80041fc

single health.py for all airflow versions

626d496

single metrics.py for all airflow versions

1c06eb2

update tests with new commons scripts

de63b71

tests: use "airflow-latest" instead of "airflow"

5e195cc

adwk67 mentioned this pull request May 15, 2025

Research Airflow 3.x support #618

Closed

6 tasks

adwk67 and others added 9 commits May 20, 2025 16:55

cleanup and code comments

999dca0

merge main and resolve conflicts

502bd19

restore deleted env

83c0394

use correct webserver service

1da6681

restore operator to release list

b97aaca

test: fix oidc

31e8c23

wip: get logging tests to work post-merge

5b772cc

make env-vars version-specific

1e105ee

fixed resolution of webserver url for execution api

714afb6

adwk67 mentioned this pull request May 23, 2025

chore(airflow): Update container images ahead of Stackable Release 25.7.0 stackabletech/docker-images#1074

Open

18 tasks

relaxed default resources

98cc63a

adwk67 mentioned this pull request May 26, 2025

chore: Bump airflow to 3.x and update docs and screenshots stackabletech/demos#218

Open

update test defs for oidc/opa

3585de5

adwk67 marked this pull request as ready for review May 27, 2025 11:12

adwk67 added this to Stackable Engineering May 27, 2025

adwk67 moved this to Development: Waiting for Review in Stackable Engineering May 27, 2025

adwk67 self-assigned this May 27, 2025

changelog

4f57856

maltesander self-requested a review May 27, 2025 12:39

maltesander moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering May 27, 2025

Merge branch 'main' into feat/airflow-3.0.0

536270f

maltesander requested changes May 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: Airflow 3.0.1 (experimental) #630

feat: Airflow 3.0.1 (experimental) #630

Uh oh!

adwk67 commented May 20, 2025 •

edited by maltesander

Loading

Uh oh!

maltesander left a comment

Uh oh!

maltesander May 27, 2025

Uh oh!

maltesander May 27, 2025

Uh oh!

Uh oh!

maltesander May 28, 2025

Uh oh!

maltesander May 28, 2025

Uh oh!

adwk67 May 28, 2025

Uh oh!

maltesander May 28, 2025

Uh oh!

Uh oh!

Uh oh!

feat: Airflow 3.0.1 (experimental) #630

Are you sure you want to change the base?

feat: Airflow 3.0.1 (experimental) #630

Uh oh!

Conversation

adwk67 commented May 20, 2025 • edited by maltesander Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

maltesander May 27, 2025

Choose a reason for hiding this comment

Uh oh!

maltesander May 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maltesander May 28, 2025

Choose a reason for hiding this comment

Uh oh!

maltesander May 28, 2025

Choose a reason for hiding this comment

Uh oh!

adwk67 May 28, 2025

Choose a reason for hiding this comment

Uh oh!

maltesander May 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adwk67 commented May 20, 2025 •

edited by maltesander

Loading